{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 2
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.6"
    },
    "pycharm": {
      "stem_cell": {
        "cell_type": "raw",
        "source": [],
        "metadata": {
          "collapsed": false
        }
      }
    },
    "colab": {
      "name": "Task 1_AML.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dhi9T8V_Z4zH",
        "colab_type": "text"
      },
      "source": [
        "# Description of the problem and solution\n",
        "\n",
        "The task1 was to predict a person's age from the brain image data: a standard regression problem. The original dataset included 832 features as well as a lot of NaN values and a few outliers. A good preprocessing stage was necessary in order to have a well defined dataset that could be used in our regression model. First step was the imputation of the dataset. Filling each NaN value with the median of each feature column. The use of the median instead of other value (e.g. mean) is justified since a lot of outliers are included in the dataset. (e.g. 1 2 _ 5 20 median: 3 mean: 7). Next step was the feature extraction. By using the \"autofeat\" library (paper: https://arxiv.org/pdf/1901.07329.pdf), we extracted the 21 most important features. The way the algorithm works is going through a loop of correlation of features with target, select promising features, train Lasso regression model with promising features, filter the good features keeping the ones with non-zero regression weights. We updated the datasets by keeping only the 21 most important features. Finally, we used these updated datasets for the training of our final regression model. A lot of outlier detection techniques were used but we decided to keep the outliers and use a tree-based method for our final model. Tree-methods have been proved to be robust to outliers and we avoid risking excluded important features / points from the dataset. The \"ExtraTreesRegressor\" model from the \"sklearn\" package was used and fine tuned based on the R2 score performance in our validation set. The final model had a score >0.6 in the validation sets using cross-validation and in the submission leaderboard of ETH scored 0.6812 while the hard baseline was set to 0.65 by the Advanced Machine Learning Task1 team."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XYdC9WtjYOBS",
        "colab_type": "text"
      },
      "source": [
        "# Include all the necessary packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "hnc7uEN97DuJ",
        "colab_type": "code",
        "outputId": "ad3c6a6b-5133-4a47-a4fe-721808514b5d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "source": [
        "!pip install autofeat\n",
        "\n",
        "from sklearn.metrics import r2_score\n",
        "\n",
        "try:\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from autofeat import FeatureSelector\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.ensemble import ExtraTreesRegressor"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: autofeat in /usr/local/lib/python3.6/dist-packages (0.2.5)\n",
            "Requirement already satisfied: pint in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.9)\n",
            "Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.25.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.21.3)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.6/dist-packages (from autofeat) (1.1.1)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.14.0)\n",
            "Requirement already satisfied: numpy in /tensorflow-2.0.0/python3.6 (from autofeat) (1.17.3)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.16.0)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24.0->autofeat) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24.0->autofeat) (2.6.1)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->autofeat) (1.3.1)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy->autofeat) (1.1.0)\n",
            "Requirement already satisfied: six>=1.5 in /tensorflow-2.0.0/python3.6 (from python-dateutil>=2.6.1->pandas>=0.24.0->autofeat) (1.12.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dAlr7AVNYVHq",
        "colab_type": "text"
      },
      "source": [
        "# Load the data from the CSV files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "Ei2HmR2e7DuQ",
        "colab_type": "code",
        "outputId": "83f32002-e4a7-4452-ab9b-d81e561e62a4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 253
        }
      },
      "source": [
        "column_names_x = ['id']\n",
        "for i in range(832):\n",
        "  column_names_x.append('x'+str(i))\n",
        "\n",
        "raw_dataset_x = pd.read_csv('/content/X_train.csv', names=column_names_x,\n",
        "                      na_values = \"?\", comment='\\t',\n",
        "                      sep=\",\", skipinitialspace=True, skiprows=True)\n",
        "\n",
        "dataset_x = raw_dataset_x.copy()\n",
        "dataset_x.tail()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>x0</th>\n",
              "      <th>x1</th>\n",
              "      <th>x2</th>\n",
              "      <th>x3</th>\n",
              "      <th>x4</th>\n",
              "      <th>x5</th>\n",
              "      <th>x6</th>\n",
              "      <th>x7</th>\n",
              "      <th>x8</th>\n",
              "      <th>x9</th>\n",
              "      <th>x10</th>\n",
              "      <th>x11</th>\n",
              "      <th>x12</th>\n",
              "      <th>x13</th>\n",
              "      <th>x14</th>\n",
              "      <th>x15</th>\n",
              "      <th>x16</th>\n",
              "      <th>x17</th>\n",
              "      <th>x18</th>\n",
              "      <th>x19</th>\n",
              "      <th>x20</th>\n",
              "      <th>x21</th>\n",
              "      <th>x22</th>\n",
              "      <th>x23</th>\n",
              "      <th>x24</th>\n",
              "      <th>x25</th>\n",
              "      <th>x26</th>\n",
              "      <th>x27</th>\n",
              "      <th>x28</th>\n",
              "      <th>x29</th>\n",
              "      <th>x30</th>\n",
              "      <th>x31</th>\n",
              "      <th>x32</th>\n",
              "      <th>x33</th>\n",
              "      <th>x34</th>\n",
              "      <th>x35</th>\n",
              "      <th>x36</th>\n",
              "      <th>x37</th>\n",
              "      <th>x38</th>\n",
              "      <th>...</th>\n",
              "      <th>x792</th>\n",
              "      <th>x793</th>\n",
              "      <th>x794</th>\n",
              "      <th>x795</th>\n",
              "      <th>x796</th>\n",
              "      <th>x797</th>\n",
              "      <th>x798</th>\n",
              "      <th>x799</th>\n",
              "      <th>x800</th>\n",
              "      <th>x801</th>\n",
              "      <th>x802</th>\n",
              "      <th>x803</th>\n",
              "      <th>x804</th>\n",
              "      <th>x805</th>\n",
              "      <th>x806</th>\n",
              "      <th>x807</th>\n",
              "      <th>x808</th>\n",
              "      <th>x809</th>\n",
              "      <th>x810</th>\n",
              "      <th>x811</th>\n",
              "      <th>x812</th>\n",
              "      <th>x813</th>\n",
              "      <th>x814</th>\n",
              "      <th>x815</th>\n",
              "      <th>x816</th>\n",
              "      <th>x817</th>\n",
              "      <th>x818</th>\n",
              "      <th>x819</th>\n",
              "      <th>x820</th>\n",
              "      <th>x821</th>\n",
              "      <th>x822</th>\n",
              "      <th>x823</th>\n",
              "      <th>x824</th>\n",
              "      <th>x825</th>\n",
              "      <th>x826</th>\n",
              "      <th>x827</th>\n",
              "      <th>x828</th>\n",
              "      <th>x829</th>\n",
              "      <th>x830</th>\n",
              "      <th>x831</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1207</th>\n",
              "      <td>1207.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5395.719279</td>\n",
              "      <td>95668.548818</td>\n",
              "      <td>1125.414599</td>\n",
              "      <td>10341.757613</td>\n",
              "      <td>10.547165</td>\n",
              "      <td>108249.187400</td>\n",
              "      <td>1.073291e+06</td>\n",
              "      <td>108672.758838</td>\n",
              "      <td>2.365013</td>\n",
              "      <td>1.042126e+06</td>\n",
              "      <td>21031.538201</td>\n",
              "      <td>1059.704248</td>\n",
              "      <td>103860.409465</td>\n",
              "      <td>107.805931</td>\n",
              "      <td>100.350289</td>\n",
              "      <td>14040.029112</td>\n",
              "      <td>NaN</td>\n",
              "      <td>90414.095308</td>\n",
              "      <td>10.108470</td>\n",
              "      <td>3.215609</td>\n",
              "      <td>102144.219482</td>\n",
              "      <td>10.087478</td>\n",
              "      <td>3825.046407</td>\n",
              "      <td>89.538601</td>\n",
              "      <td>4550.239929</td>\n",
              "      <td>3542.470382</td>\n",
              "      <td>206892.186644</td>\n",
              "      <td>1.011027e+06</td>\n",
              "      <td>11432.036864</td>\n",
              "      <td>10.957698</td>\n",
              "      <td>976.093967</td>\n",
              "      <td>84.689946</td>\n",
              "      <td>2500.095879</td>\n",
              "      <td>1068.544855</td>\n",
              "      <td>8.868114</td>\n",
              "      <td>3.695648</td>\n",
              "      <td>10381.626637</td>\n",
              "      <td>8.641068e+05</td>\n",
              "      <td>...</td>\n",
              "      <td>6.934015</td>\n",
              "      <td>2.05604</td>\n",
              "      <td>9.362337</td>\n",
              "      <td>908.498493</td>\n",
              "      <td>94.234452</td>\n",
              "      <td>10.319140</td>\n",
              "      <td>4155.345879</td>\n",
              "      <td>8.498432e+16</td>\n",
              "      <td>10977.013897</td>\n",
              "      <td>93.766125</td>\n",
              "      <td>8.120353</td>\n",
              "      <td>10595.018829</td>\n",
              "      <td>9.594486e+05</td>\n",
              "      <td>10.485803</td>\n",
              "      <td>10.601033</td>\n",
              "      <td>101328.492269</td>\n",
              "      <td>2.417968</td>\n",
              "      <td>48.359296</td>\n",
              "      <td>9486.144826</td>\n",
              "      <td>2.185578</td>\n",
              "      <td>966.094287</td>\n",
              "      <td>9.039254e+05</td>\n",
              "      <td>10.510759</td>\n",
              "      <td>1049.753163</td>\n",
              "      <td>10856.049561</td>\n",
              "      <td>9109.098919</td>\n",
              "      <td>10320.748617</td>\n",
              "      <td>10892.743222</td>\n",
              "      <td>106528.008864</td>\n",
              "      <td>294.801642</td>\n",
              "      <td>10.519831</td>\n",
              "      <td>10.234396</td>\n",
              "      <td>987.456317</td>\n",
              "      <td>978.661701</td>\n",
              "      <td>109520.061346</td>\n",
              "      <td>102914.439553</td>\n",
              "      <td>8144.701025</td>\n",
              "      <td>10.442410</td>\n",
              "      <td>102380.867791</td>\n",
              "      <td>2.236101</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1208</th>\n",
              "      <td>1208.0</td>\n",
              "      <td>93669.580198</td>\n",
              "      <td>3564.454295</td>\n",
              "      <td>96937.341346</td>\n",
              "      <td>1040.828378</td>\n",
              "      <td>8415.112792</td>\n",
              "      <td>10.721800</td>\n",
              "      <td>100628.318687</td>\n",
              "      <td>1.027671e+06</td>\n",
              "      <td>102269.472299</td>\n",
              "      <td>2.398768</td>\n",
              "      <td>1.015631e+06</td>\n",
              "      <td>10073.888339</td>\n",
              "      <td>1005.671797</td>\n",
              "      <td>106924.048275</td>\n",
              "      <td>98.512805</td>\n",
              "      <td>104.366843</td>\n",
              "      <td>11719.094168</td>\n",
              "      <td>115055.546469</td>\n",
              "      <td>95750.178334</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3.446025</td>\n",
              "      <td>100455.697718</td>\n",
              "      <td>10.303138</td>\n",
              "      <td>3502.012112</td>\n",
              "      <td>84.167435</td>\n",
              "      <td>3462.164372</td>\n",
              "      <td>3411.771480</td>\n",
              "      <td>206892.234016</td>\n",
              "      <td>1.082826e+06</td>\n",
              "      <td>9957.026155</td>\n",
              "      <td>11.155088</td>\n",
              "      <td>1127.806594</td>\n",
              "      <td>107.745872</td>\n",
              "      <td>2298.057201</td>\n",
              "      <td>1031.637383</td>\n",
              "      <td>8.778034</td>\n",
              "      <td>3.477553</td>\n",
              "      <td>10627.490011</td>\n",
              "      <td>1.007256e+06</td>\n",
              "      <td>...</td>\n",
              "      <td>10.732723</td>\n",
              "      <td>2.14034</td>\n",
              "      <td>10.930583</td>\n",
              "      <td>956.306694</td>\n",
              "      <td>117.419062</td>\n",
              "      <td>10.361670</td>\n",
              "      <td>3436.194754</td>\n",
              "      <td>6.249342e+16</td>\n",
              "      <td>9866.685225</td>\n",
              "      <td>104.420975</td>\n",
              "      <td>9.171168</td>\n",
              "      <td>10762.025617</td>\n",
              "      <td>9.205254e+05</td>\n",
              "      <td>10.627956</td>\n",
              "      <td>10.101622</td>\n",
              "      <td>105133.566646</td>\n",
              "      <td>2.740250</td>\n",
              "      <td>55.449159</td>\n",
              "      <td>10959.655054</td>\n",
              "      <td>2.160671</td>\n",
              "      <td>1111.015635</td>\n",
              "      <td>1.118527e+06</td>\n",
              "      <td>10.342601</td>\n",
              "      <td>1030.540259</td>\n",
              "      <td>10097.006670</td>\n",
              "      <td>9039.074820</td>\n",
              "      <td>10895.768148</td>\n",
              "      <td>10315.088493</td>\n",
              "      <td>106898.299599</td>\n",
              "      <td>195.245438</td>\n",
              "      <td>10.618901</td>\n",
              "      <td>10.456550</td>\n",
              "      <td>1112.699713</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>106685.647900</td>\n",
              "      <td>7428.338174</td>\n",
              "      <td>10.405107</td>\n",
              "      <td>107051.806312</td>\n",
              "      <td>2.297040</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1209</th>\n",
              "      <td>1209.0</td>\n",
              "      <td>94119.048262</td>\n",
              "      <td>NaN</td>\n",
              "      <td>88398.586879</td>\n",
              "      <td>1298.024079</td>\n",
              "      <td>8905.109893</td>\n",
              "      <td>10.294486</td>\n",
              "      <td>103347.543915</td>\n",
              "      <td>1.087571e+06</td>\n",
              "      <td>106082.959731</td>\n",
              "      <td>2.345450</td>\n",
              "      <td>1.074165e+06</td>\n",
              "      <td>49595.639929</td>\n",
              "      <td>1057.051949</td>\n",
              "      <td>103147.267707</td>\n",
              "      <td>NaN</td>\n",
              "      <td>104.099172</td>\n",
              "      <td>11275.080883</td>\n",
              "      <td>115055.523086</td>\n",
              "      <td>105201.547958</td>\n",
              "      <td>10.358605</td>\n",
              "      <td>2.872759</td>\n",
              "      <td>107834.745605</td>\n",
              "      <td>10.070086</td>\n",
              "      <td>2708.039945</td>\n",
              "      <td>90.315251</td>\n",
              "      <td>4502.083820</td>\n",
              "      <td>2267.035496</td>\n",
              "      <td>206892.199058</td>\n",
              "      <td>1.016735e+06</td>\n",
              "      <td>8545.036877</td>\n",
              "      <td>NaN</td>\n",
              "      <td>818.975938</td>\n",
              "      <td>95.255808</td>\n",
              "      <td>2614.046035</td>\n",
              "      <td>1002.028549</td>\n",
              "      <td>11.262573</td>\n",
              "      <td>3.016052</td>\n",
              "      <td>10912.480045</td>\n",
              "      <td>1.016656e+06</td>\n",
              "      <td>...</td>\n",
              "      <td>10.447547</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10.502198</td>\n",
              "      <td>967.481047</td>\n",
              "      <td>88.216016</td>\n",
              "      <td>10.535296</td>\n",
              "      <td>3548.968573</td>\n",
              "      <td>3.358649e+16</td>\n",
              "      <td>8140.800029</td>\n",
              "      <td>98.172030</td>\n",
              "      <td>11.651412</td>\n",
              "      <td>8779.078077</td>\n",
              "      <td>7.920442e+05</td>\n",
              "      <td>10.525088</td>\n",
              "      <td>10.468168</td>\n",
              "      <td>104071.834129</td>\n",
              "      <td>2.615953</td>\n",
              "      <td>130.309663</td>\n",
              "      <td>10513.170240</td>\n",
              "      <td>2.228708</td>\n",
              "      <td>728.052692</td>\n",
              "      <td>1.147362e+06</td>\n",
              "      <td>9.968195</td>\n",
              "      <td>1077.035231</td>\n",
              "      <td>7580.079750</td>\n",
              "      <td>8509.033486</td>\n",
              "      <td>10228.351982</td>\n",
              "      <td>10327.801413</td>\n",
              "      <td>100144.263391</td>\n",
              "      <td>310.183226</td>\n",
              "      <td>10.887052</td>\n",
              "      <td>10.836007</td>\n",
              "      <td>1124.755976</td>\n",
              "      <td>988.193910</td>\n",
              "      <td>102076.305244</td>\n",
              "      <td>101964.413583</td>\n",
              "      <td>6470.242459</td>\n",
              "      <td>9.786964</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2.222794</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1210</th>\n",
              "      <td>1210.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3356.949591</td>\n",
              "      <td>97081.843123</td>\n",
              "      <td>1002.778571</td>\n",
              "      <td>10864.284014</td>\n",
              "      <td>10.797471</td>\n",
              "      <td>102894.681201</td>\n",
              "      <td>NaN</td>\n",
              "      <td>100102.393811</td>\n",
              "      <td>2.436208</td>\n",
              "      <td>1.072574e+06</td>\n",
              "      <td>37587.588730</td>\n",
              "      <td>1047.896435</td>\n",
              "      <td>105241.465707</td>\n",
              "      <td>93.329187</td>\n",
              "      <td>102.255382</td>\n",
              "      <td>11668.076593</td>\n",
              "      <td>115055.525882</td>\n",
              "      <td>85619.926316</td>\n",
              "      <td>10.732048</td>\n",
              "      <td>2.422995</td>\n",
              "      <td>104044.942484</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2945.014802</td>\n",
              "      <td>112.182044</td>\n",
              "      <td>3202.074919</td>\n",
              "      <td>4883.134639</td>\n",
              "      <td>206892.208501</td>\n",
              "      <td>1.095387e+06</td>\n",
              "      <td>7036.096752</td>\n",
              "      <td>8.290676</td>\n",
              "      <td>878.350888</td>\n",
              "      <td>93.612258</td>\n",
              "      <td>2560.094596</td>\n",
              "      <td>1011.331590</td>\n",
              "      <td>10.580875</td>\n",
              "      <td>2.822595</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.085728e+06</td>\n",
              "      <td>...</td>\n",
              "      <td>10.115872</td>\n",
              "      <td>2.15810</td>\n",
              "      <td>9.868292</td>\n",
              "      <td>1004.134878</td>\n",
              "      <td>110.999990</td>\n",
              "      <td>10.713020</td>\n",
              "      <td>3394.243287</td>\n",
              "      <td>1.335970e+17</td>\n",
              "      <td>8371.821670</td>\n",
              "      <td>89.703650</td>\n",
              "      <td>9.220535</td>\n",
              "      <td>8998.055593</td>\n",
              "      <td>1.113826e+06</td>\n",
              "      <td>10.302250</td>\n",
              "      <td>10.883418</td>\n",
              "      <td>101283.417629</td>\n",
              "      <td>2.186525</td>\n",
              "      <td>366.750195</td>\n",
              "      <td>11376.166095</td>\n",
              "      <td>2.186734</td>\n",
              "      <td>1120.066965</td>\n",
              "      <td>1.013800e+06</td>\n",
              "      <td>11.734373</td>\n",
              "      <td>1046.939030</td>\n",
              "      <td>9132.087999</td>\n",
              "      <td>5896.007658</td>\n",
              "      <td>10706.113864</td>\n",
              "      <td>10369.803817</td>\n",
              "      <td>102404.821085</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10.661966</td>\n",
              "      <td>10.569144</td>\n",
              "      <td>1010.143125</td>\n",
              "      <td>1064.316139</td>\n",
              "      <td>101477.589227</td>\n",
              "      <td>104517.364548</td>\n",
              "      <td>4922.920835</td>\n",
              "      <td>8.566389</td>\n",
              "      <td>103968.235402</td>\n",
              "      <td>2.071553</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1211</th>\n",
              "      <td>1211.0</td>\n",
              "      <td>101744.439395</td>\n",
              "      <td>3416.474452</td>\n",
              "      <td>111000.425491</td>\n",
              "      <td>893.080460</td>\n",
              "      <td>9250.598106</td>\n",
              "      <td>10.935695</td>\n",
              "      <td>102906.878188</td>\n",
              "      <td>NaN</td>\n",
              "      <td>107003.164135</td>\n",
              "      <td>2.547171</td>\n",
              "      <td>1.087477e+06</td>\n",
              "      <td>32084.633800</td>\n",
              "      <td>1075.700568</td>\n",
              "      <td>104998.478525</td>\n",
              "      <td>118.678047</td>\n",
              "      <td>106.993698</td>\n",
              "      <td>13226.038641</td>\n",
              "      <td>115055.543174</td>\n",
              "      <td>97282.156471</td>\n",
              "      <td>10.208538</td>\n",
              "      <td>3.710694</td>\n",
              "      <td>107439.936587</td>\n",
              "      <td>10.753281</td>\n",
              "      <td>4719.085520</td>\n",
              "      <td>96.505582</td>\n",
              "      <td>3419.271979</td>\n",
              "      <td>3796.393911</td>\n",
              "      <td>206892.165049</td>\n",
              "      <td>1.006348e+06</td>\n",
              "      <td>11518.011808</td>\n",
              "      <td>10.803776</td>\n",
              "      <td>1103.388499</td>\n",
              "      <td>106.356520</td>\n",
              "      <td>2583.097413</td>\n",
              "      <td>1081.964569</td>\n",
              "      <td>11.453524</td>\n",
              "      <td>3.203942</td>\n",
              "      <td>10257.152912</td>\n",
              "      <td>9.292065e+05</td>\n",
              "      <td>...</td>\n",
              "      <td>8.157006</td>\n",
              "      <td>2.16880</td>\n",
              "      <td>8.844200</td>\n",
              "      <td>1005.897320</td>\n",
              "      <td>96.253172</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3289.749206</td>\n",
              "      <td>9.364744e+16</td>\n",
              "      <td>9170.136304</td>\n",
              "      <td>97.067706</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.148141e+06</td>\n",
              "      <td>10.989940</td>\n",
              "      <td>10.204516</td>\n",
              "      <td>100896.544241</td>\n",
              "      <td>2.720932</td>\n",
              "      <td>49.567830</td>\n",
              "      <td>8013.156899</td>\n",
              "      <td>2.550404</td>\n",
              "      <td>916.073375</td>\n",
              "      <td>1.080536e+06</td>\n",
              "      <td>9.101128</td>\n",
              "      <td>1013.211838</td>\n",
              "      <td>12517.066013</td>\n",
              "      <td>10990.064488</td>\n",
              "      <td>10349.678752</td>\n",
              "      <td>10096.876996</td>\n",
              "      <td>NaN</td>\n",
              "      <td>322.445715</td>\n",
              "      <td>10.599132</td>\n",
              "      <td>10.350638</td>\n",
              "      <td>962.755003</td>\n",
              "      <td>1093.734877</td>\n",
              "      <td>105353.550546</td>\n",
              "      <td>106061.798352</td>\n",
              "      <td>9302.374002</td>\n",
              "      <td>10.949418</td>\n",
              "      <td>109317.619776</td>\n",
              "      <td>2.496408</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 833 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "          id             x0           x1  ...       x829           x830      x831\n",
              "1207  1207.0            NaN  5395.719279  ...  10.442410  102380.867791  2.236101\n",
              "1208  1208.0   93669.580198  3564.454295  ...  10.405107  107051.806312  2.297040\n",
              "1209  1209.0   94119.048262          NaN  ...   9.786964            NaN  2.222794\n",
              "1210  1210.0            NaN  3356.949591  ...   8.566389  103968.235402  2.071553\n",
              "1211  1211.0  101744.439395  3416.474452  ...  10.949418  109317.619776  2.496408\n",
              "\n",
              "[5 rows x 833 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "MLVuZc0U7DuS",
        "colab_type": "code",
        "outputId": "a8b41fe5-2ee9-49ac-db56-fb5a31a3c430",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "column_names_y = ['id','y']\n",
        "raw_dataset_y = pd.read_csv('/content/y_train.csv', names=column_names_y,\n",
        "                      na_values = \"?\", comment='\\t',\n",
        "                      sep=\",\", skipinitialspace=True, skiprows=True)\n",
        "\n",
        "dataset_y = raw_dataset_y.copy()\n",
        "dataset_y.tail()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1207</th>\n",
              "      <td>1207.0</td>\n",
              "      <td>66.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1208</th>\n",
              "      <td>1208.0</td>\n",
              "      <td>73.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1209</th>\n",
              "      <td>1209.0</td>\n",
              "      <td>74.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1210</th>\n",
              "      <td>1210.0</td>\n",
              "      <td>78.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1211</th>\n",
              "      <td>1211.0</td>\n",
              "      <td>64.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          id     y\n",
              "1207  1207.0  66.0\n",
              "1208  1208.0  73.0\n",
              "1209  1209.0  74.0\n",
              "1210  1210.0  78.0\n",
              "1211  1211.0  64.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hHsZKS_XYcrh",
        "colab_type": "text"
      },
      "source": [
        "# Print the missing values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "ort_N8YQ7DuW",
        "colab_type": "code",
        "outputId": "bc80dfe0-8da4-416c-8c04-6cca8f35e5ac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "missing_values = dataset_x.isna().sum()\n",
        "print (missing_values)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "id        0\n",
            "x0       81\n",
            "x1      103\n",
            "x2       92\n",
            "x3       91\n",
            "       ... \n",
            "x827     83\n",
            "x828     78\n",
            "x829     98\n",
            "x830     84\n",
            "x831     92\n",
            "Length: 833, dtype: int64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Dpykd9sYfMj",
        "colab_type": "text"
      },
      "source": [
        "# Split the data into training and test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJDfp3ODFbrU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Split using sklearn.model_selection\n",
        "x_train, x_test, y_train, y_test = train_test_split(dataset_x, dataset_y, test_size=0.2, random_state = 100)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "iq_eFDgE7DuZ",
        "colab_type": "code",
        "outputId": "07a5f686-daab-4855-fe94-9b7d89e162a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        }
      },
      "source": [
        "train_stats = x_train.describe()\n",
        "train_stats.pop(\"id\")\n",
        "train_stats = train_stats.transpose()\n",
        "train_stats"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>mean</th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>x0</th>\n",
              "      <td>909.0</td>\n",
              "      <td>99849.359545</td>\n",
              "      <td>9534.020258</td>\n",
              "      <td>65533.368423</td>\n",
              "      <td>93818.485147</td>\n",
              "      <td>100183.062423</td>\n",
              "      <td>105994.290528</td>\n",
              "      <td>130226.576502</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x1</th>\n",
              "      <td>885.0</td>\n",
              "      <td>3698.730375</td>\n",
              "      <td>943.683864</td>\n",
              "      <td>180.312021</td>\n",
              "      <td>3076.550570</td>\n",
              "      <td>3651.110055</td>\n",
              "      <td>4303.892503</td>\n",
              "      <td>7265.213902</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x2</th>\n",
              "      <td>891.0</td>\n",
              "      <td>99975.389109</td>\n",
              "      <td>9540.065988</td>\n",
              "      <td>68544.573581</td>\n",
              "      <td>93937.346571</td>\n",
              "      <td>99386.035114</td>\n",
              "      <td>106102.200889</td>\n",
              "      <td>132221.045067</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x3</th>\n",
              "      <td>898.0</td>\n",
              "      <td>999.944996</td>\n",
              "      <td>100.903669</td>\n",
              "      <td>694.745271</td>\n",
              "      <td>935.303439</td>\n",
              "      <td>999.571797</td>\n",
              "      <td>1068.606823</td>\n",
              "      <td>1434.200505</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x4</th>\n",
              "      <td>890.0</td>\n",
              "      <td>10001.743350</td>\n",
              "      <td>1001.473353</td>\n",
              "      <td>6681.561828</td>\n",
              "      <td>9339.312428</td>\n",
              "      <td>10021.924636</td>\n",
              "      <td>10646.003276</td>\n",
              "      <td>13560.223285</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x827</th>\n",
              "      <td>906.0</td>\n",
              "      <td>104990.967566</td>\n",
              "      <td>2761.209013</td>\n",
              "      <td>100015.768596</td>\n",
              "      <td>102783.100024</td>\n",
              "      <td>104986.305216</td>\n",
              "      <td>107352.370223</td>\n",
              "      <td>109999.847537</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x828</th>\n",
              "      <td>907.0</td>\n",
              "      <td>6827.704539</td>\n",
              "      <td>1387.835714</td>\n",
              "      <td>1696.036569</td>\n",
              "      <td>6002.316474</td>\n",
              "      <td>6835.947954</td>\n",
              "      <td>7652.607118</td>\n",
              "      <td>11276.075121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x829</th>\n",
              "      <td>884.0</td>\n",
              "      <td>10.021817</td>\n",
              "      <td>0.982265</td>\n",
              "      <td>6.899008</td>\n",
              "      <td>9.378562</td>\n",
              "      <td>9.977236</td>\n",
              "      <td>10.676450</td>\n",
              "      <td>13.188278</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x830</th>\n",
              "      <td>902.0</td>\n",
              "      <td>104960.353706</td>\n",
              "      <td>2845.423469</td>\n",
              "      <td>100003.049706</td>\n",
              "      <td>102653.373914</td>\n",
              "      <td>104838.184005</td>\n",
              "      <td>107428.898901</td>\n",
              "      <td>109993.046071</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x831</th>\n",
              "      <td>893.0</td>\n",
              "      <td>2.269127</td>\n",
              "      <td>0.169559</td>\n",
              "      <td>1.589261</td>\n",
              "      <td>2.173057</td>\n",
              "      <td>2.291077</td>\n",
              "      <td>2.374205</td>\n",
              "      <td>2.846222</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>832 rows × 8 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      count           mean  ...            75%            max\n",
              "x0    909.0   99849.359545  ...  105994.290528  130226.576502\n",
              "x1    885.0    3698.730375  ...    4303.892503    7265.213902\n",
              "x2    891.0   99975.389109  ...  106102.200889  132221.045067\n",
              "x3    898.0     999.944996  ...    1068.606823    1434.200505\n",
              "x4    890.0   10001.743350  ...   10646.003276   13560.223285\n",
              "...     ...            ...  ...            ...            ...\n",
              "x827  906.0  104990.967566  ...  107352.370223  109999.847537\n",
              "x828  907.0    6827.704539  ...    7652.607118   11276.075121\n",
              "x829  884.0      10.021817  ...      10.676450      13.188278\n",
              "x830  902.0  104960.353706  ...  107428.898901  109993.046071\n",
              "x831  893.0       2.269127  ...       2.374205       2.846222\n",
              "\n",
              "[832 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mj6YSagjYlQ5",
        "colab_type": "text"
      },
      "source": [
        "# Fill the NaN in the training data set with the median values of each column"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "GSwh2Skt7Dub",
        "colab_type": "code",
        "outputId": "bf948541-3e9f-41fa-f399-3fb7ce34aeb4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "x_train = x_train.fillna(x_train.median())\n",
        "x_test = x_test.fillna(x_test.median())\n",
        "missing_values = x_train.isna().sum()\n",
        "print (missing_values)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "id      0\n",
            "x0      0\n",
            "x1      0\n",
            "x2      0\n",
            "x3      0\n",
            "       ..\n",
            "x827    0\n",
            "x828    0\n",
            "x829    0\n",
            "x830    0\n",
            "x831    0\n",
            "Length: 833, dtype: int64\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xOLIao_gYtu5",
        "colab_type": "text"
      },
      "source": [
        "# Remove the unnecessary \"id\" label"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7dPT6NsPJVsQ",
        "colab_type": "code",
        "outputId": "96e3c2b2-b0cd-4744-fafa-92842d123162",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "y_train.pop(\"id\")\n",
        "y_test.pop(\"id\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1143    1143.0\n",
              "941      941.0\n",
              "365      365.0\n",
              "467      467.0\n",
              "615      615.0\n",
              "         ...  \n",
              "156      156.0\n",
              "689      689.0\n",
              "28        28.0\n",
              "69        69.0\n",
              "1198    1198.0\n",
              "Name: id, Length: 243, dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bGDmzz2GY0xP",
        "colab_type": "text"
      },
      "source": [
        "# Feature Extraction using autofeat"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C8iEUDFdghsF",
        "colab_type": "code",
        "outputId": "e5a7f13e-0fa0-4fbf-bdc4-2531f15a824f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        }
      },
      "source": [
        "fsel = FeatureSelector(featsel_runs=4,\n",
        "        max_it=150,\n",
        "        w_thr=1e-6,\n",
        "        keep=None,\n",
        "        n_jobs=1,\n",
        "        verbose=1)\n",
        "\n",
        "new_X = fsel.fit_transform(pd.DataFrame(x_train, columns=column_names_x), y_train)\n",
        "print(new_X.columns) \n",
        "\n",
        "df_train = pd.DataFrame(x_train, columns=column_names_x)\n",
        "df_test = pd.DataFrame(x_test, columns=column_names_x)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[featsel] Scaling data...done.\n",
            "[featsel] 220/833 features after univariate filtering\n",
            "[featsel] Feature selection run 1/4\n",
            "[featsel] Feature selection run 2/4\n",
            "[featsel] Feature selection run 3/4\n",
            "[featsel] Feature selection run 4/4\n",
            "[featsel] 28 features after 4 feature selection runs\n",
            "[featsel] 28 features after correlation filtering\n",
            "[featsel] 21 features after noise filtering\n",
            "[featsel] 21 final features selected (including 0 original keep features).\n",
            "Index(['x400', 'x635', 'x757', 'x516', 'x809', 'x214', 'x556', 'x617', 'x93',\n",
            "       'x346', 'x596', 'x255', 'x309', 'x252', 'x292', 'x738', 'x537', 'x593',\n",
            "       'x474', 'x614', 'x502'],\n",
            "      dtype='object')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9pOmuH-ihJPN",
        "colab_type": "text"
      },
      "source": [
        "# Keep only the necessary features"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3A_afS8Lick8",
        "colab_type": "code",
        "outputId": "bdcab542-db46-4ff3-da1c-4428151b4826",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "dataset_selected_x = x_train.copy()\n",
        "x_test_selected= x_test.copy()\n",
        "\n",
        "#accepted=[400, 757, 635, 516, 132, 15, 809, 116, 214,\n",
        "       #556, 617, 93, 346, 596, 309, 252, 292, 474,\n",
        "       #593, 614]\n",
        "\n",
        "accepted=[400, 635, 757, 516, 809, 214, 556, 617, 93,\n",
        "       346, 596, 255, 309, 252, 292, 738, 537, 593,\n",
        "       474, 614, 502]\n",
        "\n",
        "for j in range(832):\n",
        "        if (j in accepted):\n",
        "          print (j)\n",
        "        else:\n",
        "          del dataset_selected_x['x'+str(j)]\n",
        "          del x_test_selected['x'+str(j)]\n",
        "      \n",
        "train_stats = dataset_selected_x.describe()\n",
        "train_stats.pop(\"id\")\n",
        "train_stats = train_stats.transpose()\n",
        "train_stats"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "93\n",
            "214\n",
            "252\n",
            "255\n",
            "292\n",
            "309\n",
            "346\n",
            "400\n",
            "474\n",
            "502\n",
            "516\n",
            "537\n",
            "556\n",
            "593\n",
            "596\n",
            "614\n",
            "617\n",
            "635\n",
            "738\n",
            "757\n",
            "809\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>mean</th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>x93</th>\n",
              "      <td>969.0</td>\n",
              "      <td>3.519955e+03</td>\n",
              "      <td>6.123839e+02</td>\n",
              "      <td>1.789851e+03</td>\n",
              "      <td>3.122307e+03</td>\n",
              "      <td>3.492040e+03</td>\n",
              "      <td>3.882096e+03</td>\n",
              "      <td>5.902057e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x214</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.686814e+03</td>\n",
              "      <td>4.246669e+02</td>\n",
              "      <td>3.857756e+02</td>\n",
              "      <td>1.438003e+03</td>\n",
              "      <td>1.657997e+03</td>\n",
              "      <td>1.929010e+03</td>\n",
              "      <td>3.719025e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x252</th>\n",
              "      <td>969.0</td>\n",
              "      <td>5.153136e+03</td>\n",
              "      <td>6.443209e+03</td>\n",
              "      <td>-1.066389e+04</td>\n",
              "      <td>2.036529e+03</td>\n",
              "      <td>3.051817e+03</td>\n",
              "      <td>5.396992e+03</td>\n",
              "      <td>5.218476e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x255</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.063172e+04</td>\n",
              "      <td>1.847941e+03</td>\n",
              "      <td>3.258064e+03</td>\n",
              "      <td>9.580092e+03</td>\n",
              "      <td>1.053304e+04</td>\n",
              "      <td>1.170004e+04</td>\n",
              "      <td>1.742768e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x292</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.289525e+05</td>\n",
              "      <td>1.561997e+04</td>\n",
              "      <td>7.115248e+04</td>\n",
              "      <td>1.201669e+05</td>\n",
              "      <td>1.282218e+05</td>\n",
              "      <td>1.377145e+05</td>\n",
              "      <td>1.982293e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x309</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.360934e+04</td>\n",
              "      <td>2.104294e+03</td>\n",
              "      <td>1.787475e+03</td>\n",
              "      <td>1.241801e+04</td>\n",
              "      <td>1.355888e+04</td>\n",
              "      <td>1.480791e+04</td>\n",
              "      <td>2.142239e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x346</th>\n",
              "      <td>969.0</td>\n",
              "      <td>7.264234e+03</td>\n",
              "      <td>1.238998e+03</td>\n",
              "      <td>2.195927e+03</td>\n",
              "      <td>6.577637e+03</td>\n",
              "      <td>7.355575e+03</td>\n",
              "      <td>8.085946e+03</td>\n",
              "      <td>1.121581e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x400</th>\n",
              "      <td>969.0</td>\n",
              "      <td>2.419177e+00</td>\n",
              "      <td>1.566607e-01</td>\n",
              "      <td>1.441218e+00</td>\n",
              "      <td>2.325897e+00</td>\n",
              "      <td>2.416256e+00</td>\n",
              "      <td>2.507563e+00</td>\n",
              "      <td>3.029658e+00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x474</th>\n",
              "      <td>969.0</td>\n",
              "      <td>2.138704e+05</td>\n",
              "      <td>3.363320e+04</td>\n",
              "      <td>6.580233e+04</td>\n",
              "      <td>1.951336e+05</td>\n",
              "      <td>2.113702e+05</td>\n",
              "      <td>2.311579e+05</td>\n",
              "      <td>4.824998e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x502</th>\n",
              "      <td>969.0</td>\n",
              "      <td>7.341196e+13</td>\n",
              "      <td>5.051213e+13</td>\n",
              "      <td>-8.384285e+13</td>\n",
              "      <td>4.129257e+13</td>\n",
              "      <td>6.245895e+13</td>\n",
              "      <td>9.198935e+13</td>\n",
              "      <td>3.816907e+14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x516</th>\n",
              "      <td>969.0</td>\n",
              "      <td>2.633410e+00</td>\n",
              "      <td>2.851324e-01</td>\n",
              "      <td>1.586758e+00</td>\n",
              "      <td>2.461447e+00</td>\n",
              "      <td>2.632617e+00</td>\n",
              "      <td>2.810227e+00</td>\n",
              "      <td>3.709460e+00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x537</th>\n",
              "      <td>969.0</td>\n",
              "      <td>2.131250e+05</td>\n",
              "      <td>3.401347e+04</td>\n",
              "      <td>6.499225e+04</td>\n",
              "      <td>1.936266e+05</td>\n",
              "      <td>2.105931e+05</td>\n",
              "      <td>2.305157e+05</td>\n",
              "      <td>4.810740e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x556</th>\n",
              "      <td>969.0</td>\n",
              "      <td>6.058210e+03</td>\n",
              "      <td>7.982839e+02</td>\n",
              "      <td>3.040824e+03</td>\n",
              "      <td>5.588225e+03</td>\n",
              "      <td>5.998493e+03</td>\n",
              "      <td>6.472923e+03</td>\n",
              "      <td>9.103862e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x593</th>\n",
              "      <td>969.0</td>\n",
              "      <td>6.306444e+04</td>\n",
              "      <td>3.517018e+04</td>\n",
              "      <td>2.381309e+04</td>\n",
              "      <td>4.825502e+04</td>\n",
              "      <td>5.195309e+04</td>\n",
              "      <td>5.625009e+04</td>\n",
              "      <td>2.232860e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x596</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.994849e+04</td>\n",
              "      <td>2.631193e+03</td>\n",
              "      <td>9.693873e+03</td>\n",
              "      <td>1.845468e+04</td>\n",
              "      <td>1.980071e+04</td>\n",
              "      <td>2.136755e+04</td>\n",
              "      <td>3.037075e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x614</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.219718e+06</td>\n",
              "      <td>1.863793e+05</td>\n",
              "      <td>4.090431e+05</td>\n",
              "      <td>1.112420e+06</td>\n",
              "      <td>1.220616e+06</td>\n",
              "      <td>1.319038e+06</td>\n",
              "      <td>1.976749e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x617</th>\n",
              "      <td>969.0</td>\n",
              "      <td>1.528155e+03</td>\n",
              "      <td>6.897489e+02</td>\n",
              "      <td>-6.478282e+02</td>\n",
              "      <td>1.039647e+03</td>\n",
              "      <td>1.425202e+03</td>\n",
              "      <td>1.899843e+03</td>\n",
              "      <td>5.028253e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x635</th>\n",
              "      <td>969.0</td>\n",
              "      <td>2.552238e+00</td>\n",
              "      <td>2.211141e-01</td>\n",
              "      <td>1.536237e+00</td>\n",
              "      <td>2.441985e+00</td>\n",
              "      <td>2.571545e+00</td>\n",
              "      <td>2.685138e+00</td>\n",
              "      <td>3.319348e+00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x738</th>\n",
              "      <td>969.0</td>\n",
              "      <td>4.070505e+05</td>\n",
              "      <td>5.419321e+04</td>\n",
              "      <td>1.500802e+05</td>\n",
              "      <td>3.777463e+05</td>\n",
              "      <td>4.051612e+05</td>\n",
              "      <td>4.389639e+05</td>\n",
              "      <td>6.508670e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x757</th>\n",
              "      <td>969.0</td>\n",
              "      <td>2.556026e+00</td>\n",
              "      <td>2.138400e-01</td>\n",
              "      <td>1.513769e+00</td>\n",
              "      <td>2.448038e+00</td>\n",
              "      <td>2.577740e+00</td>\n",
              "      <td>2.684761e+00</td>\n",
              "      <td>3.292584e+00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>x809</th>\n",
              "      <td>969.0</td>\n",
              "      <td>8.301919e+01</td>\n",
              "      <td>1.026332e+02</td>\n",
              "      <td>-1.034091e+02</td>\n",
              "      <td>3.267047e+01</td>\n",
              "      <td>5.403747e+01</td>\n",
              "      <td>9.901100e+01</td>\n",
              "      <td>1.245880e+03</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      count          mean  ...           75%           max\n",
              "x93   969.0  3.519955e+03  ...  3.882096e+03  5.902057e+03\n",
              "x214  969.0  1.686814e+03  ...  1.929010e+03  3.719025e+03\n",
              "x252  969.0  5.153136e+03  ...  5.396992e+03  5.218476e+04\n",
              "x255  969.0  1.063172e+04  ...  1.170004e+04  1.742768e+04\n",
              "x292  969.0  1.289525e+05  ...  1.377145e+05  1.982293e+05\n",
              "x309  969.0  1.360934e+04  ...  1.480791e+04  2.142239e+04\n",
              "x346  969.0  7.264234e+03  ...  8.085946e+03  1.121581e+04\n",
              "x400  969.0  2.419177e+00  ...  2.507563e+00  3.029658e+00\n",
              "x474  969.0  2.138704e+05  ...  2.311579e+05  4.824998e+05\n",
              "x502  969.0  7.341196e+13  ...  9.198935e+13  3.816907e+14\n",
              "x516  969.0  2.633410e+00  ...  2.810227e+00  3.709460e+00\n",
              "x537  969.0  2.131250e+05  ...  2.305157e+05  4.810740e+05\n",
              "x556  969.0  6.058210e+03  ...  6.472923e+03  9.103862e+03\n",
              "x593  969.0  6.306444e+04  ...  5.625009e+04  2.232860e+05\n",
              "x596  969.0  1.994849e+04  ...  2.136755e+04  3.037075e+04\n",
              "x614  969.0  1.219718e+06  ...  1.319038e+06  1.976749e+06\n",
              "x617  969.0  1.528155e+03  ...  1.899843e+03  5.028253e+03\n",
              "x635  969.0  2.552238e+00  ...  2.685138e+00  3.319348e+00\n",
              "x738  969.0  4.070505e+05  ...  4.389639e+05  6.508670e+05\n",
              "x757  969.0  2.556026e+00  ...  2.684761e+00  3.292584e+00\n",
              "x809  969.0  8.301919e+01  ...  9.901100e+01  1.245880e+03\n",
              "\n",
              "[21 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmYercpLg9ZC",
        "colab_type": "text"
      },
      "source": [
        "# Regression model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "name": "#%%\n",
          "is_executing": false
        },
        "id": "GKd9O-Hu7Duv",
        "colab_type": "code",
        "outputId": "cbc67610-a56a-4723-9b53-689fa57eea88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "rfr = ExtraTreesRegressor(n_jobs=1, max_depth=None, n_estimators=180, random_state=0, min_samples_split=3, max_features=None)\n",
        "\n",
        "rfr.fit(dataset_selected_x, np.ravel(y_train))\n",
        "\n",
        "y_pred = rfr.predict(x_test_selected) \n",
        "\n",
        "score = r2_score(y_test, y_pred)\n",
        "print(score)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.6037280104368132\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0CERsJNeg4Ct",
        "colab_type": "text"
      },
      "source": [
        "# Export the file with our predictions for submission"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nmDyNBr4kSra",
        "colab_type": "code",
        "outputId": "923ef6a9-b849-4980-c8bc-407a0a5b9f5b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "#accepted=[400, 757, 635, 516, 132, 15, 809, 116, 214,\n",
        "       #556, 617, 93, 346, 596, 309, 252, 292, 474,\n",
        "       #593, 614]\n",
        "\n",
        "accepted=[400, 635, 757, 516, 809, 214, 556, 617, 93,\n",
        "       346, 596, 255, 309, 252, 292, 738, 537, 593,\n",
        "       474, 614, 502]\n",
        "\n",
        "column_names_x = ['id']\n",
        "for i in range(832):\n",
        "  column_names_x.append('x'+str(i))\n",
        "\n",
        "raw_dataset_x_test = pd.read_csv('/content/X_test.csv', names=column_names_x,\n",
        "                      na_values = \"?\", comment='\\t',\n",
        "                      sep=\",\", skipinitialspace=True, skiprows=True)\n",
        "\n",
        "dataset_x_test = raw_dataset_x_test.copy()\n",
        "dataset_x_test.tail()\n",
        "\n",
        "dataset_x_test = dataset_x_test.fillna(dataset_x_test.median())\n",
        "\n",
        "dataset_selected_x_test = dataset_x_test.copy()\n",
        "\n",
        "for j in range(832):\n",
        "        if (j in accepted):\n",
        "          print (j)\n",
        "        else:\n",
        "          del dataset_selected_x_test['x'+str(j)]\n",
        "\n",
        "predictions = rfr.predict(dataset_selected_x_test)\n",
        "\n",
        "index = 0.0\n",
        "with open('predictions.txt', 'w') as f:\n",
        "    f.write(\"%s\\n\" % \"id,y\")\n",
        "    for predict in predictions:\n",
        "        writing_str = str(index)+','+str(predict.item(0))\n",
        "        f.write(\"%s\\n\" % writing_str)\n",
        "        index = index + 1"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "93\n",
            "214\n",
            "252\n",
            "255\n",
            "292\n",
            "309\n",
            "346\n",
            "400\n",
            "474\n",
            "502\n",
            "516\n",
            "537\n",
            "556\n",
            "593\n",
            "596\n",
            "614\n",
            "617\n",
            "635\n",
            "738\n",
            "757\n",
            "809\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}